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(54) Image coding apparatus and method, image decoding apparatus and method, and recording 
medium 



(57) This invention is intended to prevent a mis- 
match from occurring between a re-encoded and re- 
corded video stream and other data. A transport stream 
inputted in a recording apparatus is supplied to a demul- 
tiplexer and an analyzing block. The analyzing block 
outputs non-video stream information to a coding con- 
troller. The coding controller controls the encoding op- 
eration of an encoder on the basis of the inputted infor- 
mation and outputs video coding control information to 



a coding block. On the basis of video coding control in- 
formation, the coding block generates multimedia dis- 
play sub information (including information about a pic- 
ture frame for example) which prevents a mismatch from 
occurring between video and other data on the repro- 
ducing side and outputs this information to a writing 
block. The writing block records the video stream and 
other data to a recording medium as well as the multi- 
media display sub information generated by the coding 
block. 
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BACKGROUND OF THE INVENTION 

[0001] The present invention relates generally to an 
image coding apparatus and method, an image decod- 
ing apparatus and method, and a recording medium. 
More specifically, but not exclusively, the present inven- 
tion relates to an image coding apparatus and method, 
an image decoding apparatus and method, and a re- 
cording medium which are suitably for use in apparatus- 
es for re-encoding video streams and recording and re- 
producing the re-encoded video streams. 
[0002] Digital television broadcasts such as Europe- 
an DVB (Digital Video Broadcast), American DTV (Dig- 
ital Television) broadcast, and Japanese BS (Broadcast 
Satellite) digital broadcast use MPEG (Moving Picture 
Expert Group) 2 transport streams. A transport stream 
consists of continuous transport packets, each packet 
carrying video data or audio data for example. The data 
length of one transport packet is 1 88 bytes. 
[0003] Unlike the analog television broadcast, the dig- 
ital television broadcast is capable of providing services 
added with multimedia coding data. In these services, 
data such as video data, audio data, character graphics 
data, and still picture data for example are associated 
with each other by the multimedia coding data for trans- 
mission. For the multimedia coding data, a coding meth- 
od based on XML (Extensible Markup Language) is 
used in the Japanese BS digital broadcast for example. 
The details of this method is disclosed in ARIB STD-B24 
Data Coding And Transmission Specification for Digital 
Broadcasting, for example. 

[0004] The data such as video data, audio data, char- 
acter graphics data, and still picture data are each pack- 
etized into a transport packet for transmission. 
[0005] Now, referring to FIGS. 1 A and 1 B, an example 
of synthesizing data to be transferred between the send- 
ing and receiving sides and a multimedia screen. As 
shown in FIG. 1 A, the sending side sends to the receiv- 
ing side video data, character graphics data for display- 
ing buttons A through C, text data for displaying "XYZ- 
ABC...," and multimedia coding data for relating these 
data each other. The sending side generally denotes a 
television broadcast station for example. However, it 
herein denotes a television broadcast station which in- 
cludes a recording apparatus (the recording side) which 
receives and records data transmitted from broadcast 
stations, the example illustrated in FIG. 1 A including the 
data which is outputted from this recording apparatus. 
[0006] The multimedia coding data includes data 
which can synthesize on the receiving side video data, 
character graphics data, and text data and display the 
synthesized data. To be more specific, the multimedia 
coding data includes the data associated with the dis- 
play positions of the video, character graphics, and text 
which are displayed by the size-associated data such 
as multimedia plane (the display area of images on the 



television receiver for exampUlJ size (plane_height and 
plane_width) and video display size (video_height and 
video_width), video data, character graphics data, and 
text data as shown in FIG. 1B. 
5 [0007] On the basis of the multimedia coding data, the 
receiving side processes the video data, the character 
graphics data, and the text data to display a resultant 
image as shown in FIG. 1B. 

[0008] Through the screen on which the above-men- 

10 tioned image is displayed, the user can receive services 
such as displaying desired information in the video sec- 
tion by clicking button A corresponding to that informa- 
tion and obtaining, from the text data displayed in the 
bottom of the screen, the information associated with a 

15 matter displayed in the video section, for example. 
[0009] If a television program carried by a transport 
stream transmitted from a digital television broadcast is 
recorded without changed to a recording medium on the 
received side, the program can be recorded without its 

20 picture and audio qualities being deteriorated at all. 
However, in order to record a program as long as pos- 
sible to a recording medium of a limited recording ca- 
pacity by presupposing a certain degree of picture qual- 
ity deterioration, the received video stream must be de- 

25 coded and then encoded again to lower the bit rate of 
the transport stream. 

[0010] For example : the re-encoding of the video 
stream of a television program attached with multimedia 
coding data to lower its bit rate for recording may be 

30 implemented by sub-sampling the image to change writ- 
ing blocks. However, this approach presents a problem 
of causing a mismatch in the relationship between the 
video stream resulted from re-encoding and the multi- 
media coding data. The following describes an example 

35 of this mismatch with reference to FIGS. 2A and 2B. 
[0011] In the example shown in FIG. 2A, the sending 
side (the recording side) converts the original video writ- 
ing block to a smaller picture frame at the time of re- 
encoding. Therefore, as shown in FIG. 2B, on the re- 

40 ceiving side (the reproducing side), changes occur in 
video display size and position, resulting in a display 
screen which is different from the display screen intend- 
ed by the sending side (the display screen to be dis- 
played on the basis of the data before being re-encod- 

45 ed). 

SUMMARY OF THE INVENTION 

[0012] Various aspects and features of the present in- 
50 vention are defined in the appended claims. 

[0013] According to a first aspect, there is provided a 
first image coding apparatus comprising: inputting 
means for inputting a multiplexed stream containing 
multimedia coding data; separating means for separat- 
55 ing a video stream from the multiplexed stream inputted 
by the inputting means; converting means for perform- 
ing a predetermined converting process on the video 
stream separated by the separating means; generating 
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means for generating additional infoMEtion indicating 
that a mismatch will occur when the converted video 
stream is displayed on the basis of the multimedia cod- 
ing data; and outputting means for outputting the con- 
verted video stream, the multimedia coding data, and 
the additional information. 

[001 4] Embodiments of the present invention can pro- 
vide an image coding apparatus and method, an image 
decoding apparatus and method, and a recording me- 
dium, wherein information that no mismatch is caused 
or at least the possibility of mismatch is reduced in the 
relationship between a video stream after re-encoding 
and other data is generated and recorded on the record- 
ing side, and wherein it is prevented a shrunk picture 
frame, for example, from occurring on the reproducing 
side by referencing the information generated on the re- 
cording side. 

[0015] The first image coding apparatus may further 
comprise: coding means for coding the additional infor- 
mation generated by the generating means as data sep- 
arate from the multiplexed stream containing the con- 
verted video stream. 

[0016] The first image coding apparatus may further 

comprise: coding means for multiplexing the additional 

information generated by the generating means with the 

multiplexed stream containing the converted video 

stream and then coding a multiplexed result. 

[0017] The converting means may convert a video 

stream picture frame parameter. 

[0018] The conversion by the converting means may 

include at least a process of decoding the video stream 

separated by the separating means and a process of 

encoding the decoded video stream. 

[0019] The additional information generated by the 

generating means may contain at least one of original 

picture frame information and an original screen aspect 

ratio. 

[0020] The additional information generated by the 
generating means may contain an original video format 
and a video format after the conversion. 
[0021] The additional information generated by the 
generating means may contain an original screen as- 
pect ratio and a screen aspect ratio after the conversion. 
[0022] The additional information generated by the 
generating means may contain at least one of informa- 
tion indicating whether or not a picture frame of the video 
stream has been converted by the converting means, 
information about an original picture frame of the video 
stream separated by the separating means, and an orig- 
inal screen aspect ratio. 

[0023] In carrying out the invention and according to 
a second aspect thereof, there is provided a first image 
coding method comprising the steps of: inputting a mul- 
tiplexed stream containing multimedia coding data; sep- 
arating a video stream from the multiplexed stream in- 
putted in the inputting step; performing a predetermined 
converting process on the video stream separated in the 
separating step; generating additional information indi- 



cating that a mismatch will oRTjr when the converted 
video stream is displayed on the basis of the multimedia 
coding data; and outputting the converted video stream , 
the multimedia coding data, and the additional informa- 
5 tion. 

[0024] The first image coding method may further 
comprise the step of: coding the additional information 
generated in the generating step as data separate from 
the multiplexed stream containing the converted video 
10 stream. 

[0025] The first image coding method may further 
comprise: 

a coding step of multiplexing the additional information 
generated in the generating step with the multiplexed 
is stream containing the converted video stream and then 
coding a multiplexed result. 

[0026] A video stream picture frame parameter may 

be converted in the converting step. 

[0027] The conversion in the converting step may in- 

20 elude at least a process of decoding the video stream 
separated in the separating step and a process of en- 
coding the decoded video stream. 
[0028] The additional information generated in the 
generating step may contain at least one of original pic- 

25 ture frame information and an original screen aspect ra- 
tio. 

[0029] The additional information generated in the 
generating step may contain an original video format 
and a video format after the conversion. 

30 [0030] The additional information generated in the 
generating step may contain an original screen aspect 
ratio and a screen aspect ratio after the conversion. 
[0031] The additional information generated in the 
generating step may contain at least one of information 

35 indicating whether or not a picture frame of the video 
stream has been converted in the converting step, in- 
formation about an original picture frame of the video 
stream separated in the separating step, and an original 
screen aspect ratio. 

40 [0032] In carrying out the invention and according to 
a third aspect thereof, there is provided a first recording 
medium recorded a computer-readable program, the 
program comprising the steps of: inputting a multiplexed 
stream containing multimedia coding data; separating a 

45 video stream from the multiplexed stream inputted in the 
inputting step; performing a predetermined converting 
process on the video stream separated in the separating 
step; generating additional information indicating that a 
mismatch will occur when the converted video stream 

50 is displayed on the basis of the multimedia coding data; 
and outputting the converted video stream, the multime- 
dia coding data, and the additional information. 
[0033] In carrying out the invention and according to 
a fourth aspect thereof, there is provided a second re- 

55 cording medium recorded a video stream converted by 
a predetermined conversion process, multimedia cod- 
ing data, and additional information indicating occur- 
rence of a mismatch at displaying the converted video 
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stream on the basis of the multimeo>BK>ding data. 
[0034] The additional information may be coded and 
recorded as data different from a multiplexed stream 
containing the converted video stream. 
[0035] The additional information may be coded and 
recorded as multiplexed with a multiplexed stream con- 
taining the converted video stream. 
[0036] The video stream may be converted in its pic- 
ture frame parameter. 

[0037] The video stream may be decoded and then 
encoded. 

[0038] The additional information may contain at least 
one of original picture frame information and an original 
screen aspect ratio. 

[0039] The additional information may contain infor- 
mation about an original video format and information 
about a video format after the conversion. 
[0040] The additional information may contain infor- 
mation about an original screen aspect ratio and infor- 
mation about a screen aspect ratio after the conversion, 
[0041] The additional information may contain at least 
one of information indicating whether or not a picture 
frame of the video stream has been converted, informa- 
tion about an original picture frame of the video stream, 
and information about an original screen aspect ratio. 
[0042] In carrying out the invention and according to 
a fifth aspect thereof, there is provided a first image de- 
coding apparatus comprising: inputting means for input- 
ting a multiplexed stream containing multimedia coding 
data; separating means for separating a video stream 
from the multiplexed stream inputted by the inputting 
means; decoding means for decoding the video stream 
separated by the separating means; and processing 
means for performing a predetermined conversion proc- 
ess on the decoded video stream in accordance with 
additional information indicating occurrence of a mis- 
match at displaying the decoded video stream on the 
basis of the multimedia decoding data. 
[0043] The first image decoding apparatus may fur- 
ther comprise: acquiring means for acquiring the addi- 
tional information from data different from the multi- 
plexed stream. 

[0044] The first image decoding apparatus may fur- 
ther comprise: acquiring means for acquiring the addi- 
tional information from a multiplexed stream with which 
the additional information is multiplexed. 
[0045] The processing means may convert a picture 
frame parameter of the video stream. 
[0046] The conversion by the processing means may 
include at least a process of decoding the video stream 
separated by the separating means and a process of 
encoding the decoded video stream. 
[0047] The additional information may contain at least 
one of information about an original picture frame and 
information about an original screen aspect ratio. 
[0048] The additional information may contain an 
original video format and information about a video for- 
mat after the conversion. 
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[0049] The additional infoWTation may contain infor- 
mation about an original screen aspect ratio and infor- 
mation about a screen aspect ratio after the conversion. 
[0050] The additional information may contain at least 

5 one of information indicating whether or not a picture 
frame of the video stream has been converted by the 
converting means, information about an original picture 
frame of the video stream separated by the separating 
means, and information about an original screen aspect 

10 ratio. 

[0051] In carrying out the invention and according to 
a sixth aspect thereof, there is provided a first image 
decoding method comprising the steps of: inputting a 
multiplexed stream containing multimedia coding data; 
15 separating a video stream from the multiplexed stream 
inputted in the inputting step; decoding the video stream 
separated in the separating step; and performing a pre- 
determined conversion process on the decoded video 
stream in accordance with additional information indi- 
go eating occurrence of a mismatch at displaying the de- 
coded video stream on the basis of the multimedia de- 
coding data. 

[0052] The first image decoding method may further 
comprise the step of: acquiring the additional informa- 

25 tion from data different from the multiplexed stream. 
[0053] The first image decoding method may further 
comprise the step of: acquiring the additional informa- 
tion from a multiplexed stream with which the additional 
information is multiplexed. 

30 [0054] A picture frame parameter of the video stream 
may be converted in the processing step. 
[0055] The conversion in the processing step may in- 
clude at least a process of decoding the video stream 
separated in the separating step and a process of en- 

35 coding the decoded video stream. 

[0056] The additional information may contain at least 
one of information about an original picture frame and 
information about an original screen aspect ratio. 
[0057] The additional information may contain an 

40 original video format and information about a video for- 
mat after the conversion. 

[0058] The additional information may contain infor- 
mation about an original screen aspect ratio and infor- 
mation about a screen aspect ratio after the conversion. 

45 [0059] The additional information may contain at least 
one of information indicating whether or not a picture 
frame of the video stream has been converted in the 
converting step, information about an original picture 
frame of the video stream separated in the separating 

50 step, and information about an original screen aspect 
ratio. 

[0060] In carrying out the invention and according to 
a seventh aspect thereof there is provided a third re- 
cording medium recorded a computer-readable pro- 
55 gram, the program comprising the steps of: inputting a 
multiplexed stream containing multimedia coding data; 
separating a video stream from the multiplexed stream 
inputted in the inputting step; decoding the video stream 
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separated in the separating step; arURforming a pre- 
determined conversion process on the decoded video 
stream in accordance with additional information indi- 
cating occurrence of a mismatch at displaying the de- 
coded video stream on the basis of the multimedia de- 5 
coding data. 

[0061] In carrying out the invention and according to 
an eighth aspect thereof, there is provided a second im- 
age coding apparatus comprising: inputting means for 
inputting a multiplexed stream; separating means for 10 
separating a video stream from the multiplexed stream 
inputted by the inputting means; determining means for 
determining whether or not multimedia coding data is 
contained in the multiplexed stream inputted by the in- 
putting means; generating means, if it is determined by is 
the determining means that the multimedia coding data 
is contained in the multiplexed stream, for generating 
coding control information for instructing not to change 
a display format of the video stream separated by the 
separating means; converting means for performing a 20 
predetermined conversion process on the video stream 
separated by the separating means on the basis of the 
coding control information generated by the generating 
means; and multiplexing means for generating a multi- 
plexed stream which contains the video stream convert- 25 
ed by the converting means. 

[0062] The generating means may instruct not to 
change any of a picture frame, a video format, and an 
aspect ratio. 

[0063] In carrying out the invention and according to 30 
a ninth aspect thereof, there is provided a second image 
coding method comprising the steps of: inputting a mul- 
tiplexed stream; separating a video stream from the mul- 
tiplexed stream inputted in the inputting step; determin- 
ing whether or not multimedia coding data is contained 35 
in the multiplexed stream inputted in the inputting step; 
generating coding control information for instructing not 
to change a display format of the video stream separat- 
ed in the separating step, if it is determined in the deter- 
mining step that the multimedia coding data is contained 40 
in the multiplexed stream; performing a predetermined 
conversion process on the video stream separated in 
the separating step on the basis of the coding control 
information generated in the generating step; and gen- 
erating a multiplexed stream which contains the video 45 
stream converted in the converting step. 
[0064] Coding control information for instructing not 
to change any of a picture frame, a video format, and 
an aspect ratio may be generated in the generating step. 
[0065] In carrying out the invention and according to so 
a tenth aspect thereof, there is provided a fourth record- 
ing medium recorded a computer-readable program, 
the program comprising the steps of: inputting a multi- 
plexed stream; separating a video stream from the mul- 
tiplexed stream inputted in the inputting step; determin- 
ing whether or not multimedia coding data is contained 
in the multiplexed stream inputted in the inputting step; 
generating coding control information for instructing not 
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to change a display format dfflTe video stream separat- 
ed in the separating step, if it is determined in the deter- 
mining step that the multimedia coding data is contained 
in the multiplexed stream; performing a predetermined 
conversion process on the video stream separated in 
the separating step on the basis of the coding control 
information generated in the generating step; and gen- 
erating a multiplexed stream which contains the video 
stream converted in the converting step. 
[0066] In carrying out the invention and according to 
an eleventh aspect thereof, there is provided a fifth re- 
cording medium recorded coding control information in- 
structing not to change a display format of a video 
stream and a multiplexed stream containing a video 
stream on which a predetermined conversion process 
has been performed on the basis of the coding control 
information. 

[0067] As described and according to the first image 
coding apparatus and method and the program stored 
in the first recording medium, a video steam is separated 
from a multiplexed stream containing multimedia coding 
data, and a predetermined conversion process is per- 
formed on the separated video stream, and additional 
information indicating that a mismatch will occur when 
displaying the converted video stream on the basis of 
the multimedia coding data. 

[0068] The second recording medium stores a video 
stream converted by a predetermined conversion proc- 
ess, the multimedia coding data, and the additional in- 
formation indicating that a mismatch will occur when dis- 
playing the converted video stream on the basis of the 
above-mentioned multimedia coding data. 
[0069] As described and according to the image de- 
coding apparatus and method and the program stored 
in the third recording medium, when a video stream is 
separated from an inputted multiplexed stream, the sep- 
arated video stream is decoded, and the decoded video 
stream is displayed on the basis of multimedia coding 
data, a mismatch occurs. On the basis of the additional 
information about this mismatch occurrence, a prede- 
termined conversion process is performed on the de- 
coded video stream. This novel configuration prevents 
the mismatch from occurring between the video stream 
and the multimedia coding data. 

[0070] As described and according to the second im- 
age coding apparatus and method and the program 
stored in the fourth recording medium, a video stream 
is separated from an inputted multiplexed stream, the 
inputted multiplexed stream is checked whether or not 
multimedia coding data is contained and, if the multime- 
dia coding data is contained, coding control information 
for giving an instruction not to change the display format 
of the separated video stream is generated, and a pre- 
determined conversion process is performed on the 
separated video stream on the basis of the generated 
coding control information. 

[0071] The fifth recording medium also stores the 
above-mentioned coding control information giving in- 
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struction not to change the displa^BFhat of a video 
stream and a multiplexed stream containing the video 
stream on which a predetermined conversion process 
has been performed on the basis of the coding control 
information. 5 

BRIEF DESCRIPTION OF THE DRAWINGS 
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[0072] The invention will now be described by way of 
example with reference to the accompanying drawings, 
throughout which like parts are referred to by like refer- 
ences, and in which: 



FIGS. 1A and 1B are schematic diagrams illustrat- 
ing a display screen to be shown on the basis of 15 
multimedia coding information; 
FIGS. 2A and 2B are schematic diagrams illustrat- 
ing a mismatch which takes place when a video 
stream is re-encoded; 

FIG. 3 is a block diagram illustrating a recording ap- 20 
paratus practiced as one embodiment of the 
present invention; 

FIGS. 4A and 4B illustrate an operation of a multi- 
plexer shown in FIG. 3; 

FIGS. 5A, 5B and 5C illustrate the processing by an 25 
arrival timestamp adding block; 
FIG. 6 illustrates multimedia display sub informa- 
tion; 

FIG. 7 illustrates an example of Programlnfo() syn- 
tax; 30 
FIG. 8 illustrates an example of StreamCodinglnfo 
() syntax; 

FIG. 9 illustrates the meaning of stream_coding 
type; 

FIG. 1 0 illustrates the meaning of video_format; 35 
FIG. 11 illustrates the meaning of frame_rate; 
FIG. 12 illustrates the meaning of 
display_aspect_ratio; 

FIG. 13 is a flowchart describing the processing of 
coding AV stream and multimedia display sub infor- 40 
mation; 

FIG. 14 is a flowchart describing the coding 
processing to be executed for restricting the re-en- 
coding of a multiplexed stream video including mul- 
timedia coding data; 45 
FIG. 15 illustrates an example of an input transport 
stream; 

FIG. 1 6 illustrates an example of a transport stream 
after the re-encoding of the video stream shown in 
FIG. 15; 50 
FIG. 17 is a flowchart describing a recording rate 
control process by a recording apparatus shown in 
FIG. 3; 

FIG. 18 is a flowchart describing another recording 
rate control process by the recording apparatus 55 
shown in FIG. 3; 

FIG. 19 illustrates another example of a transport 
stream resulted from the re-encoding of the video 



stream; 

FIG. 20 illustrates another example of the input 
transport stream; 

FIG. 21 is a block diagram illustrating a configura- 
tion of a reproducing apparatus practiced as one 
embodiment of the present invention; 
FIGS. 22A and 22B illustrate a display screen to be 
shown when multimedia display sub information is 
added; 

FIG. 23 is a block diagram illustrating another con- 
figuration of the recording apparatus practiced as 
one embodiment of the present invention; 
FIG. 24 is a flowchart describing the processing of 
reproducing an AV stream which uses multimedia 
display sub information; 

FIG. 25 is a block diagram illustrating another con- 
figuration of the reproducing apparatus practiced as 
one embodiment of the present invention; and 
FIG. 26 illustrates recording media. 

DETAILED DESCRIPTION OF PREFERRED 
EMBODIMENTS 

[0073] This invention will be described in further detail 
by way of example with reference to the accompanying 
drawings. Now, referring to FIG. 3, there is shown a 
block diagram illustrating an exemplary configuration of 
a recording apparatus 1 practiced as one embodiment 
of the invention. A transport stream received at an an- 
tenna, not shown, is inputted in a selector 1 0. A program 
number (a channel number) specified by the user is also 
inputted from a terminal 11 to the selector 10. Referring 
to the received program number, the selector 1 0 ex- 
tracts the specified program from the received transport 
stream and outputs a partial transport stream. The par- 
tial transport stream is inputted in a demultiplexer 1 2 and 
an analyzing block 13. 

[0074] The partial transport stream inputted in the de- 
multiplexer 1 2 is separated into a video stream and oth- 
er streams (audio, still picture, character graphics, and 
multimedia coding data for example). The video stream 
thus obtained is outputted to a decoder 14. The other 
streams are outputted to a multiplexer 1 6. In addition to 
the transport packets other than video, the demultiplex- 
er 12 outputs the output timing information in the input 
transport stream of these transport packets to the mul- 
tiplexer 1 6. 

[0075] The decoder 14 applies a predetermined de- 
coding scheme, for example, MPEG2 to the inputted 
video stream and outputs the decoded video data to an 
encoder 15. Also, the decoder 14 outputs the stream 
information about the video stream obtained at decod- 
ing to a coding controller 1 8. 

[0076] On the other hand, the analyzing block 1 3 an- 
alyzes the inputted transport stream to obtain the stream 
information about the non-video streams, for example, 
a bit rate and outputs it to the coding controller 18. The 
stream information about the non-video streams output- 
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ted from the analyzing block 13, the^BB stream infor- 
mation outputted from decoder 1 4, and a stream record- 
ing bit rate outputted from a terminal 19 are inputted in 
the coding controller 18. From these data, the coding 
controller 1 8 sets the video data coding conditions (cod- 5 
ing control information) to be executed by the encoder 
15 and outputs these coding conditions to the encoder 
15 and a coding block 20. 

[0077] The coding controller 18 uses, as a bit rate to 
be allocated to the video data encoding, a value ob- 10 
tained by subtracting a total value (the data inputted 
from the analyzing block 13) of the bit rates of the non- 
video streams from a stream recording bit rate (the data 
inputted, via the terminal 19, from a controller, not 
shown, for controlling the operation of the recording ap- is 
paratus 1 , for example). The coding controller 18 sets 
coding control information such as bit rate and picture 
frame such that an optimum picture quality can be 
achieved with the bit rate thus obtained and outputs this 
coding control information to the encoder 15 and the 20 
coding block 20. The details of the coding control infor- 
mation will be described later with reference to FIGS. 
15 through 20. 

[0078] When a stream is recorded to a recording me- 
dium with a fixed rate, this stream recording bit rate be- 25 
comes the fixed rate; if a stream is recorded with a var- 
iable bit rate, this stream recording bit rate is a mean bit 
rate per predetermined time. However, the maximum 
value of the variable bit rate in this case needs to be 
lower than the maximum recording bit rate ensured by 30 
the recording medium concerned. 
[0079] The encoder 15 encodes (on the basis of 
MPEG2 for example) the video data outputted from the 
decoder 14 on the basis of the coding control informa- 
tion outputted from the coding controller 1 8 and outputs 35 
the resultant video data to the multiplexer 1 6. The video 
stream from the encoder 15, the transport stream pack- 
ets other than video from the demultiplexer 12, and the 
information about the occurrence timing of the transport 
stream packets otherthan video are inputted in the mul- 40 
tiplexer 1 6. On the basis of the inputted occurrence tim- 
ing information, the multiplexer 1 6 multiplexes the video 
stream with the transport stream packets other than vid- 
eo and outputs the result to the arrival timestamp adding 
block 1 7 as a transport stream. 45 
[0080] FIGS. 4A and 4B schematically illustrate the 
above-mentioned processing to be executed by the mul- 
tiplexer 16. FIG. 4A shows the timing of the inputted 
transport stream packets. In these figures, the cross- 
hatched portions indicate the video packets while the so 
white portions indicate the stream packets other than 
video. As shown in FIG. 4A, the inputted transport 
stream packets are continuous; however, the data vol- 
ume of the video data is reduced by the re-encoding of 
video data by the encoder 15. Consequently, the 55 
number of video packets is reduced. 
[0081] As shown in FIG. 4B, the multiplexer 16 does 
not change the timing of the stream packets other than 
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video but causes only the tirrvrhy&f the video packets to 
be different from the original state (shown in FIG. 4A). 
[0082] As shown in FIGS. 5A, 5B and 5C. the arrival 
timestamp adding block 17 adds a header 
(TP_extra_header) including an arrival timestamp to 
each of the packets (FIG. 5A) of the inputted transport 
stream to generate a source packet (FIG. 5B), arranges 
the generated source packets continuously (FIG. 5C), 
and outputs them to a writing block 21 . The arrival times- 
tamp is information indicative of the timing with which 
the transport stream packets occur in a transport 
stream. The writing block 21 puts the inputted source 
packet stream consisting of continuous source packets 
and records the file to a recording medium 22. It should 
be noted that the recording medium 22 may be any type 
of recording medium. 

[0083] The information outputted from the coding 
block 20 is also inputted in the writing block 21 . On the 
basis of the video coding information from the coding 
controller 1 8, the coding block 20 generates multimedia 
display sub information and outputs the same to the writ- 
ing block 21 . The multimedia display sub information to 
be outputted to the writing block 21 is information for 
keeping the video display position and display size un- 
changed on multimedia plane from those of the image 
(the image which would be displayed without re-encod- 
ing) intended by the sending side even if the picture 
frame size has changed by transcoding (decoding by 
the decoder 14 and then encoding by the encoder 15) 
a video stream. This information is also used at time of 
reproduction in combination with multimedia coding da- 
ta. 

[0084] The following describes the mu ltimedia display 
sub information more specifically. As shown in FIG. 6, 
the multimedia display sub information consists of three 
flags of a mismatch flag (mismatch Jv1Minfo_f lag), a re- 
encoded flag (Re_encoded_flag), and a frame size 
change flag (changed_frame_size_flag), data associat- 
ed with two sizes indicative of an original horizontal size 
(original_horizontal_size) and an original vertical size 
(original_verticaLsize), and an original screen aspect 
ratio (original_display_aspect_ratio). 
[0085] The mismatch flag indicates whether there ex- 
ists a mismatch in the relationship between video and 
multimedia coding data. The re-encoded flag indicates 
whether the video has been re-encoded at the time of 
recording. The frame size change flag indicates whether 
the picture frame of video has been changed by re-en- 
coding for example. The original horizontal size indi- 
cates the horizontal size of a picture frame before re- 
encoding. The original vertical size indicates the vertical 
size of a picture frame before re-encoding. The original 
screen aspect ratio indicates the aspect ratio of a frame 
screen before re-encoding. 

[0086] It should be noted that the above-mentioned 
multimedia display sub information is illustrative only. 
Therefore, information other than shown in FIG. 6 may 
be included in or part of the information shown in FIG. 
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6 may be excluded from the multimeWFdisplay sub in- 
formation. 

[0087] The following describes another example of 
the multimedia display sub information. In the following 
example, the multimedia display sub information is 5 
stored in a Programing) syntax shown in FIG. 7. The 
following describes the fields associated with the 
present invention in the Programing) syntax. 
[0088] "length" indicates the number of bytes be- 
tween the byte just afterthe length field and the last byte 10 
of Programlnfo() inclusive. 

[0089] "num_of_program_sequences" indicates the 
number of program sequences in the an AV stream file. 
A source packet sequence with which the program con- 
tents specified by this format in the AV stream file are is 
constant is referred to as a program sequence. 
[0090] "SPM_program_sequence_start" indicates an 
address at which the program sequence starts in the AV 
stream file. "SPM_program_sequence_start" is of a size 
in unit of source packet number and counted from the 20 
initial value 0 starting with the first packet of the AV 
stream file. 

[0091] "program_map_PID" is value of the PID of a 
transport packet having PMT (Program Map Table) ap- 
plicable to that program sequence. 25 
[0092] ,, num_of_streams_in_ps" . indicates the 
number of elementary streams defined in that program 
sequence. 

[0093] "stream_PID" indicates the value of the PID for 
the elementary stream defined in the PMT which is ref- 30 
erenced by the program map PID of that program se- 
quence. 

[0094] "StreamCodinglnfoO" indicates the informa- 
tion about the elementary stream indicated by the 
above-mentioned stream PID. 35 
[0095] FIG. 8 shows the syntax of StreamCodinglnfo 
(). "length" indicates the number of bytes between the 
byte just after this length field and the last byte of 
StreamCodinglnfoO inclusive. 

[0096] "stream_coding_type" indicates the coding 40 
type of the elementary stream indicated by the stream 
PID for this StreamCodinglnfoO. The meanings of the 
individual types are shown in FIG. 9. 
[0097] If the value of stream coding type is 0x02, it 
indicates that the elementary stream indicated by the 45 
stream PID is a video stream. 

[0098] If the value of stream coding type is OxOA, 
OxOB, or OxOD, it indicates that the elementary stream 
indicated by the stream PID is multimedia coding data. 
[0099] If the value of stream coding type is 0x06, it 50 
indicates that the elementary stream indicated by the 
stream PID is subtitles or teletext. 
[0100] ,, video_format" indicates the video format of a 
video stream indicated by the stream PID for this 
StreamCodingl nf o(). The meanings of the individual vid- 55 
eo formats are shown in FIG. 1 0. 

[0101] In FIG. 10, 480i indicates video display of NT- 
SC standard TV (interlace frame of 720 pixels x 480 
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lines). 576i indicates video aTSJtfay of PAL standard TV 
(interlace frame of 720 pixels x 576 lines). 480p indi- 
cates video display of progressive frame of 720 pixels x 
480 lines. 1080i indicates video display of interlace 
frame of 1920 pixels x 1080 lines. 720p indicates video 
display of progressive frame of 1230 pixels x 720 lines. 
[01 02] "frame_rate" indicates the frame rate of a video 
stream indicated by the stream PID for this StreamCod- 
inglnfoO. The meanings of the individual frame rates are 
shown in FIG. 11. 

[0103] M display_aspct_ratio" indicates the display as- 
pect ratio of a video stream indicated by the stream PID 
for this StreamCodinglndo(). The meaning of the indi- 
vidual aspect ratios are shown in FIG. 12. 
[0104] "original_video_format_flag" indicates wheth- 
er there exist original video format and original display 
aspect ratio in this StreamCodinglnfoO- 
[0105] "original_video_format" indicates a video for- 
mat before a video stream indicated by the stream PID 
for this StreamCodinglnfoO 's coded. The meanings of 
the individual original video formats are the same as 
shown in FIG. 10. 

[0106] "originaLdisplay_aspect_ratio" is the display 
aspect ratio before a video stream indicated by the 
stream PID for this StreamCodinglnfoO is coded. The 
meanings of the individual aspect ratios are the same 
as shown in FIG. 12. 

[0107] It is assumed that, in transcoding a transport 
stream with a multimedia data stream (BML stream or 
subtitles) multiplexed along with a video stream, the re- 
encoding of the video stream changes its video format 
(for example, from 1080i to 480i), while the multimedia 
data stream retain its original stream contents. In this 
case, a mismatch in information may occur between a 
new video stream and the multimedia data stream. For 
example, although the parameters associated with the 
display of the multimedia data stream are determined 
on the supposition of the video format of the original vid- 
eo stream, the video format may be changed by the re- 
encoding of the video stream. 

[0108] The video format of the original video stream 
is indicated by the video format and the display aspect 
ratio. The video format of the re-encoded video stream 
is indicated by the original video format and the original 
display aspect ratio. 

[01 09] If there exists a mismatch between the values 
of the video format and the original video format and/or 
between the display aspect ratio and the original display 
aspect ratio, it indicates that a video format change has 
been caused by the video re-encoding at the time of re- 
cording. 

[0110] If the stream PID in which the stream coding 
type indicates multimedia coding data and subtitles is 
included in Program lnfo() : it indicates that the multime- 
dia data is multiplexed in an AV stream file (a transport 
stream). 

[0111] If Programlnfo() indicates that a video format 
change has been caused by the re-encoding of video at 
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the time of recording and multimedilWWa is multiplexed 
in the AV stream file, then it is determined that there ex- 
ists a mismatch in display between the video stream (re- 
encoded) and the multimedia data (the original multime- 
dia data) in the AV stream file. s 
[01 12] In such a case, the information about the orig- 
inal video stream, namely the original video format and 
the original display aspect ratio, becomes effective. The 
reproducing apparatus generates a display screen from 
the above-mentioned new video stream and multimedia 10 
data stream as follows. 

The video stream is up-sampled to a video format 
indicated by the original video format and the orig- 
inal display aspect ratio. 15 
The up-sampled image and the multimedia data 
stream are synthesized to form a correct display 
screen. 

[01 13] The multimedia display sub information gener- 20 
ated by the coding block 20 is recorded by the writing 
block 21 to the recording medium 22 but stored as a file 
which is different from the source packet stream file out- 
putted from the arrival timestamp adding block 1 7. If the 
multimedia display sub information is recorded by the 25 
writing block 21 to the recording medium 22 as a file 
different from the source packet stream file, the filed 
multimedia display sub information is outputted from the 
coding block 20. 

[0114] FIG. 13 is a flowchart describing the process- 30 
ing of coding an AV stream and multimedia display sub 
information. 

[01 1 5] In step 50, a multiplexed stream including mul- 
timedia coding data is inputted in the recording appara- 
tus 1 . 35 
[0116] In step 51 , the demultiplexer 12 separates the 
video stream from the multiplexed stream. 
[0117] In step 52, the encoder 1 5 re-encodes the vid- 
eo stream decoded by the decoder 1 4. 
[0118] In step 53, the multiplexer 16 multiplexes the 40 
above-mentioned video stream and multimedia coding 
data to generate a multiplexed stream. 
[0119] In step 54, the coding block 20 generates mul- 
timedia display sub information. 

[0120] In the above description, the coding controller 45 
1 8 generates the coding control information including bit 
rate and picture frame on the basis of the inputted data. 
The coding controller 1 8 may generate the following in- 
formation as alternative coding control information. 
Namely, if the inputted transport stream is found includ- so 
ing multimedia coding data by the analyzing block 13, 
then the coding controller 18 may generate coding con- 
trol information, when encoding is executed by the en- 
coder 15, for instructing the encoder 15 to execute the 
re-encoding with a picture frame (the picture frame be- 55 
fore re-encoding) of the same size as that of the picture 
frame of the original video and output the generated 
coding control information to the encoder 15. 
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[0121] When the above- nfieTvtioned method is used, 
the encoder 1 5 re-encodes the video data supplied from 
the decoder 1 4 with the same value as that of the picture 
frame of the original video stream on the basis of the 
inputted coding control information. If such coding con- 
trol information is generated and the re-encoding is ex- 
ecuted on the basis of the coding control information, 
no picture frame change is caused by the re-encoding, 
thereby preventing a mismatch from occurring in the re- 
lationship between the video stream obtained by re-en- 
coding and the multimedia coding data. 
[0122] Still alternatively, the following information may 
be generated as the coding control information gener- 
ated by the coding controller 1 8. Namely, if the inputted 
transport stream is found including multimedia coding 
data by the analyzing block 13, then the coding control- 
ler 18 may generate coding control information, when 
encoding is executed by the encoder 15, for instructing 
the encoder 1 5 to execute the re-encoding under the 
same conditions as the video format (shown in FIG. 1 0) 
and screen aspect ratio (shown in FIG. 12) of the original 
video and output the coding control information to the 
encoder 15. 

[0123] When the above-mentioned method is used, 
the encoder 15 re-encodes the video supplied from the 
decoder 1 4 under the same conditions as the video for- 
mat (shown in FIG. 10) and screen aspect ratio (shown 
in FIG. 12) of the original video on the basis of the in- 
putted coding control information. If such coding control 
information is generated and the re-encoding is execut- 
ed on the basis of the coding control information, no vid- 
eo format and no screen aspect ratio change is caused 
by the re-encoding, thereby preventing a mismatch from 
occurring in the relationship between the video stream 
obtained by re-encoding and the multimedia coding da- 
ta. 

[0124] FIG. 14 is a flowchart describing the coding for 
restricting the re-encoding of the video of a multiplexed 
stream including multimedia coding data. 
[0125] In step 70, a multiplexed stream is inputted in 
the recording apparatus 1 . 

[0126] In step 71 , the demultiplexer 12 separates the 
video stream from the multiplexed stream. 
[0127] In step 72, the analyzing block 1 3 checks if the 
multimedia coding data is included in the video stream. 
If the multimedia coding data is included, the analyzing 
block 13 sends the coding control information to the en- 
coder 15 instructing the same to re-encode the video 
stream without changing the display format. On the ba- 
sis of the supplied control information, the encoder 15 
re-encodes the video stream. 

[0128] In step 73, the multiplexer 1 6 generates a mul- 
tiplexed stream including the above-mentioned video 
stream. 

[0129] The following describes one example of con- 
trol to be executed on the basis of the coding control 
information with reference to FIGS. 15 through 20. 
[0130] It is assumed here that a transport stream to 
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be inputted in the selector 1 0 have aSWffant bit rate R, 
as shown in FIG. 15 for example. The video stream and 
the non-video streams are coded by variable bit rates. 
In the example shown in FIG. 1 5, in unit time (for exam- 
ple, GOP) A, the bit rate of the video stream is R VA and 
the bit rate of non-video streams is R OA . In unit time B, 
the bit rate of the video stream is R VB and the bit rate of 
non-video streams is R OB . In unit time C, the bit rate of 
the video stream is R vc and the bit rate of non-video 
streams is R oc . 

[0131] If the transport stream as shown in FIG. 15 is 
re-encoded to output the transport stream having fixed 
bit rate S (S < R,) as shown in FIG. 16 from the multi- 
plexer 16, the coding controller 18 executes the 
processing described by thefiowchart shown in FIG. 1 7. 
[0132] First, in step S1 , the coding controller 18 sets 
to S the bit rate (recording rate) of a transport stream to 
be outputted from the multiplexer 16 on the basis of a 
control signal inputted from a controller, not shown, via 
the terminal 19. Next, in step S2, the coding controller 
1 8 determines non-video streams to be recorded and 
computes a maximum total value D of the bit rates of 
the determined streams. 

[0133] The maximum value D is determined from the 
stream specification of the input transport stream. For 
example, if two audio streams are to be recorded in ad- 
dition to the video stream, the maximum value D is 384 
x 2 Kbps since the maximum value of the bit rate of one 
audio stream is 384 Kbps according to the Japanese 
digital BS broadcast stream specification. 
[0134] Instep S3, the coding controller 18 uses value 
C obtained by subtracting the maximum value D com- 
puted in step S2 from the recording bit rates set in step 
S1 (= S - D), as a bit rate to be allocated to the re-en- 
coding of the video data. In step S4, the coding controller 
18 analyzes the coding information such as the video 
stream bit rate and picture frame from the video stream 
information outputted from the decoder 14. 
[0135] In step S5, the coding controller 18 deter- 
mines, on the basis of the value C computed in step S3 
and the video stream coding information analyzed in 
step S4, a video coding parameter (video coding control 
information) such that an optimum picture quality is 
achieved. 

[0136] For example, in the example shown in FIG. 16, 
value S is 1/2 of value R,. In the present example, the 
bit rate of steams other than video is the maximum value 
D, which is used without change as the bit rate of non- 
video steams in a multiplexed stream after re-encoding. 
[0137] Then, video coding parameters are deter- 
mined such that an optimum picture quality can be 
achieved within the range of (S - D). If the picture frame 
is controlled, the horizontal direction of a picture frame 
of 720 x 480 pixels for example is sampled by 1/2 into 
360 x 480 pixels. The determined coding parameters (bit 
rate and picture angle) are supplied to the encoder 15 
as video coding control information. 
[0138] In step S6, on the basis of the video coding 
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control information supplied IWffi the coding controller 
18, the encoder 15 re-encodes the video data of unit 
time (in this example, unit time A) to be processed now. 
In the example shown in FIG. 1 6, the actual bit rate R OA 

5 is smaller than the maximum value D in unit time A; how- 
ever, since the maximum value D is fixed, the video al- 
located bit rate becomes (S - D). A wasted portion Rsa 
which cannot be used for video coding occurs because 
the maximum value D is fixed, the wasted portion being 

10 filled with stuffing bits. 

[0139] In stepS7, the coding controller 18 determines 
whether there remains any stream to be re-encoded. If 
there still remains any stream to be re-encoded, the pro- 
cedure returns to step S4 to repeat the above-men- 

15 tioned processes. 

[0140] If, in step S7, there remains no more stream to 
be re-encoded, this processing comes to an end. 
[0141] Thus, in the example shown in FIG. 16, the bit 
rate of non-video streams is D also in unit time B and 

20 the video stream allocated bit rate is S - D because it is 
fixed. Stuffing bits are inserted in value R sb (= S - (S - 
D) - R OB = D - R OB ). 

[0142] In unit time C, too, the bit rate of non-video 
streams is D and the video stream allocated bit rate is 
25 S - D. It should be noted that, in unit time C, D = R oc , so 
that there exists no stuffing bits. 

[0143] Thus, in the example shown in FIG. 1 6, the vid- 
eo stream is coded with a fixed bit rate. 
[0144] FIG. 18 is a flowchart describing a processing 

30 example in which the video re-encoding allocated bit 
rate is variable. First, in step S21 , the coding controller 
18 sets recording rate S on the basis of the information 
supplied via the terminal 19. Next, in step S22, the cod- 
ing controller 18 analyzes the coding information of the 

35 video stream on the basis of the video stream informa- 
tion supplied from the decoder 14. The processes of 
steps S21 and S22 are the same as those of steps S1 
and S4 of FIG. 17. 

[01 45] In step S23, the coding controller 1 8 computes, 
40 from the output of the analyzing block 13, the total bit 
rate B in each unit time of non-video streams. 
[0146] In step S24, the coding controller 18 uses, as 
the video re-encoding allocated bit rate, value C (= S - 
B) obtained by subtracting value B obtained in step S23 
45 from value S obtained in S1 . 

[0147] In step S25, the coding controller 18 deter- 
mines, on the basis of value C obtained in step S24 and 
a result of analysis of the video stream coding informa- 
tion obtained in step S22, video coding parameters such 
50 that an optimum picture quality is obtained. The deter- 
mined coding parameters are outputted to the encoder 
15. 

[0148] In step S26, the encoder 15 re-encodes the 
video data of the current unit time on the basis of the 
55 coding parameters determined in step S25. Conse- 
quently, as shown in FIG. 19 for example, after alloca- 
tion of R oa (=R OA ) as the bit rate in unit time of non-video 
streams, the bit rate of the video stream is set to bit rate 
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[0149] In step S27, the coding controller 18 deter- 
mines whether there remains any stream to be proc- 
essed. If there still remains any stream to be processed, 
the procedure returns to step S22 to repeat the above- 5 
mentioned processes. If there remains no more stream 
to be processed, this processing comes to an end. 
[0150] Thus, in unit time B, after allocation of bit rate 
R ob (= S - R OB ) of non-video streams, the remaining 
(= S - R ob ) is the bit rate of the video stream. In unit time 10 
C, the bit rate of the video stream is set to R vc (= S - R^) 
except for bit rate Roc of non-video streams. 
[0151] Thus, in the present processing example, the 
bit rate of the video stream is variable and therefore no 
stuffing bit is needed or the number of stuffing bits can 15 
be reduced, thereby coding the video stream more effi- 
ciently. 

[0152] In the above, the input transport stream has a 
fixed bit rate. The present invention is also applicable to 

r - an example in which the bit rate of the input transport 20 

' stream is variable as shown in FIG. 20. 

[0153] Consequently, a transport stream of longer 
content can be recorded to the recording medium 22 at 
a lower bit rate as required. 

[0154] In addition, the above-mentioned novel consti- 25 
tution prevents the qualities of audio data, still picture 
and character graphics data, multimedia coding data, 
and other non-video data from being conspicuously de- 
teriorated. These non-video data is basically smaller in 
data volume than video data, so that reducing their bit 30 
rate in the same ratio as the bit rate of video data makes 
the effects on these non-video data relatively greater 
than those on video data. The novel constitution can 
prevents these effects from being caused. 
[0155] The following describes the reproduction of a 35 
source packet stream file recorded on the recording me- 
dium 22. Referring to FIG. 21, there is shown a block 
diagram illustrating the configuration of a reproducing 
apparatus practiced as one embodiment of the inven- 
(T~ \ tion. A source packet stream file recorded on the record- 40 

ing medium 22 is read by a reading block 31 . The read- 
ing block 31 also reads multimedia display sub informa- 
tion recorded on the recording medium 22 as a file sep- 
arate from the source packet stream file. 
[01 56] The source packet stream read by the reading 
block 31 is outputted to a arrival timestamp separating 
block 32 and the multimedia display sub information to 
a synthesizing block 36. The arrival timestamp separat- 
ing block 32 incorporates a reference clock. The arrival 
time stamp separating block 32 compares the reference so 
clock with the value of the arrival timestamp added to 
the source packet of the inputted source packet stream 
and, when a match is found, removes the arrival times- 
tamp from the source packet having the matching arrival 
timestamp, outputting the resultant packet to a demulti- 55 
plexer 33 as a transport stream packet. 
[0157] The demultiplexer 33 separates the inputted 
transport stream into a video/audio stream and data 
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streams such as multimedia coding data, character 
graphics, text, and still picture. Of these separated data, 
the video/audio stream is outputted to an AV decoder 
34, the multimedia coding data to the synthesizing block 
36, and the data stream such as character graphics, 
text, and still picture to a character graphics/still picture 
decoder 35. 

[01 58] The AV decoder 34 separates the inputted vid- 
eo/audio stream into video data and audio data, de- 
codes each data, and outputs the decoded audio data 
to an audio reproducing device, not shown, and the de- 
coded video data to the synthesizing block 36 . The char- 
acter graphics/still picture decoder 35 decodes the in- 
putted data stream such as character graphics, text, and 
still picture and outputs the decoded character graphics 
data, text data, and still picture data to the synthesizing 
block 36. 

[0159] In the synthesizing block 36, the video data 
from the AV decoder 34, the multimedia coding data 
from the demultiplexer 33, the data from the character 
graphics/still picture decoder 35, and the multimedia 
display sub information from the reading block 31 are 
inputted. Checking the mismatch flag (FIG. 6) of the in- 
putted multimedia display sub information, the synthe- 
sizing block 36 determines whether there exits a mis- 
match in the relationship between the inputted video sig- 
nal and the multimedia coding data. 
[0160] If there exists a mismatch between the value 
of video format and the value of original video format 
shown in FIG. 8 and/or a mismatch between the value 
of display aspect ratio and the original display aspect 
ratio, the synthesizing block 36 determines that a video 
format change has been caused by the video re-encod- 
ing at the time of recording, detecting a mismatch in the 
relationship between the input video signal and the mul- 
timedia encoding data. If there exists no mismatch be- 
tween the value of video format and the value of original 
video format and no mismatch between the value of dis- 
play aspect radio and the value of original display aspect 
ratio, the synthesizing block 36 determines that there 
exists no mismatch in the relationship between the input 
video signal and the multimedia coding data. 
[0161] If a mismatch is found in the relationship be- 
tween the input video signal and the multimedia coding 
data, the synthesizing block 36 further references the 
original horizontal size and vertical size of the multime- 
dia display sub information or references the original 
video format and the original display aspect ratio. Then, 
the synthesizing block 36 scale-converts the inputted 
video signal so that it can be displayed in a frame of the 
referenced size. On the basis of the multimedia coding 
data, the synthesizing block 36 outputs the video signal 
with the scale-converted video signal and the data such 
as character graphics synthesized on a multimedia 
plane to a television receiver, not shown, which servers 
as a display device. 

[0162] On the other hand, if no mismatch is found in 
the relationship between the input video signal and the 
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multimedia coding data, the syntheslll^f block 36 syn- 
thesizes the input video signal with other data on a mul- 
timedia plane without scale conversion and outputs the 
synthesized data. 

[0163] Thus, recording the multimedia display sub in- 
formation and using it at the time of reproduction allow 
the receiving side to display a screen as intended on the 
sending side. Referring to FIG. 22, if the re-encoding on 
the sending side (recording side) results in a smaller vid- 
eo picture frame than the original, the size reduction is 
recorded as multimedia display sub information, which 
is referenced at the time of reproduction. Consequently, 
because there exists no mismatch between video data 
and other data, the receiving side (the reproduction 
side) can display the same screen as the original. 
[0164] FIG. 24 is a flowchart describing AV stream re- 
production processing which uses multimedia display 
sub information. 

[0165] In step 60, a multiplexed stream including mul- 
timedia coding data is read from a recording medium 
and inputted in a reproduction device. 
[01 66] In step 61 , multimedia display sub information 
is inputted. This information is read from the recording 
medium in the case of the reproducing device shown in 
FIG. 21 ; in the case of a reproducing device shown in 
FIG. 25 : this information is separated from the multi- 
plexed stream. 

[0167] In step 62, a video stream is separated from 
the multiplexed stream. 

[0168] In step 63, the video stream is decoded. 
[0169] In step S64, if there exists a mismatch is dis- 
play between the video data and the multimedia coding 
data, the synthesizing block 36 scale-converts the video 
data on the basis of the multimedia display sub informa- 
tion. 

[0170] In step 65, the synthesizing block 36 synthe- 
sizes the processed image and the multimedia data to 
generate a display image. 

[01 71 ] As described, the multimedia display sub infor- 
mation may be recorded on the recording medium 2 as 
a file which is different from the source packet stream 
file containing character graphics data and video sig- 
nals. Alternatively, the mutlimedia display sub informa- 
tion may be embedded in a source packet stream file 
and then recorded on the recording medium 22. FIG. 23 
shows the configuration of the recording apparatus 1 in 
which the multimedia display sub information is embed- 
ded in a source packet stream file. 
[0172] In comparison between the configuration of the 
recording apparatus 1 shown in FIG. 23 and the config- 
uration shown in FIG. 3, the former outputs the multime- 
dia display sub information outputted from the coding 
block 20 is supplied to the multiplexer 16. The multiplex- 
er 16 then generates a transport packet of the inputted 
multimedia display sub information embeds it into a 
source packet stream file, outputting the same to the ar- 
rival timestamp adding block 1 7. Instead of embedding 
the multimedia display sub information into a source 
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packet stream file as a transpW^acket, the multimedia 
display sub information may be written to a user data 
area in an MPEG video stream. 

[0173] In the present embodiment of the invention, 
5 video data may be re-encoded in other methods than 
that described above; for example, an inputted video 
stream may be converted in the DCT area to covert the 
coding parameters such as picture frame. 
[0174] FIG. 25 shows the configuration of the repro- 
10 ducing apparatus 30 in which the multimedia display sub 
information is embedded in a source packet stream file 
to be recorded on the recording medium 22. In compar- 
ison between the configuration of the reproducing ap- 
paratus shown in FIG. 25 and the configuration shown 
*5 in FIG. 21 . the former reads through the reading block 
31 only the source packet stream. The source packet 
stream read by the reading block 31 is inputted in the 
demultiplexer 33 via the arrival timestamp separating 
block 32. 

20 [0175] The demultiplexer 33 extracts the multimedia 
display sub information from the inputted source packet 
stream file and outputs the extracted information to the 
synthesizing block 36. The further processing is the 
same as that of the configuration shown in FIG. 5. 

25 [0176] Thus, if the multimedia display sub information 
is recorded as embedded in a source packet stream file, 
the receiving side can also obtain the video picture size 
and display position intended by the sending side. 
[0177] In the present embodiment of the invention, a 

30 transport stream was used for example. The present in- 
vention is also applicable to multiplexed streams such 
as a program stream. 

[0178] The above-described sequence of processing 
operations can be executed by hardware as well as soft- 

35 ware. In the software approach, the recording apparatus 
1 (and the reproducing apparatus 30) is constituted by 
a personal computer as shown in FIG. 26. 
[0179] Referring to FIG. 26, a CPU (Central Process- 
ing Unit) 101 executes various processing operations 

40 as instructed by programs stored in a ROM (Read Only 
Memory) 1 02 or loaded from a storage block 1 08 into a 
RAM (Random Access Memory) 103. The RAM 103 al- 
so stores, as required, the data necessary for the CPU 
101 to execute various processing operations. 

45 [0180] The CPU 101, the ROM 1 02, and the RAM 1 03 
are interconnected via a bus 104. The bus 104 is also 
connected to an input/output interface 105. 
[0181] The input/output interface 1 05 is connected to 
an input block 1 06 such as a keyboard and a mouse, a 

so display device such as CRT or LCD : an output block 1 07 
such as a speaker, a storage block 108 such as hard 
disk, and a communication block 109 such as modem 
or terminal adapter. The communication block 109 exe- 
cutes communication processing via a network. 

55 [01 82] The input/output interface 1 05 is also connect- 
ed to a drive 110 as required, in which a magnetic disc 
121 , an optical disc 122, a magneto-optical disc 123, or 
a semiconductor memory 124 is loaded. Computer pro- 
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grams read from these storage mell^re installed in 
the storage block 108 as required. 
[01 83] The execution of a sequence of processing op- 
erations by software requires the use of a computer hav- 
ing a dedicated hardware device storing beforehand the 5 
programs constituting the software or a general-pur- 
pose computer in which these programs are installed as 
required from a recording medium. 
[0184] The program recording medium for storing 
computer-readable and executable programs may be a 10 
package medium which is distributed to users providing 
programs and constituted by the magnetic disk 121 or 
(including floppy disk), the optical disc 122 (including 
CD-ROM (Compact Disc-Read Only Memory) and DVD 
(Digital Versatile Disc)), the magneto-optical disk 123 15 
(including MD (Mini Disk)), or the semiconductor mem- 
ory 1 24 or a ROM 1 02 or a hard disk which is preinstalled 
in a personal computer and provided for users and on 
which the programs are stored temporarily or perma- 
nently as shown in FIG. 26. 20 
[01 85] It should be noted that the steps describing the 
programs to be stored in the program storage medium 
are not only executed in a time-dependent manner in 
the order described, but also in parallel or in a discrete 
manner. 25 
[0186] As described and according to the first image 
coding apparatus and method and the program stored 
in the first recording medium, a video steam is separated 
from a multiplexed stream containing multimedia coding 
data, a predetermined conversion process is performed 30 
on the separated video stream, and additional informa- 
tion indicative that a mismatch will occur when display- 
ing the converted video stream on the basis of the mul- 
timedia coding data. 

[0187] The first recording medium stores the convert- 35 
ed video stream, the multimedia coding data, and the 
additional information indicative that a mismatch will oc- 
cur when displaying the converted video stream on the 
basis of the above-mentioned multimedia coding data. 
[0188] Consequently, in any case, the reproducing 40 
side can prevent a mismatch from occurring between 
the video stream and the multimedia coding data. 
[0189] As described and according to the image de- 
coding apparatus and method and the program stored 
in the second recording medium, when a video stream 45 
is separated from an inputted multiplexed stream, the 
separated video stream is decoded, and the decoded 
video stream is displayed on the basis of multimedia 
coding information, a mismatch occurs. On the basis of 
the additional information about this mismatch occur- so 
rence, a predetermined conversion process is per- 
formed on the decoded video stream. This novel con- 
figuration prevents the mismatch from occurring be- 
tween the video stream and the multimedia coding data. 
[0190] As described and according to the second im- 55 
age coding apparatus and method and the program 
stored in the third recording medium, a video stream is 
separated from an inputted multiplexed stream, the in- 
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putted multiplexed stream is"c>iecked for multimedia 
coding data and, if the multimedia coding data is found, 
coding control information for giving an instruction not 
to change the display format of the separated video 
stream is generated, and a predetermined conversion 
process is performed on the separated video stream on 
the basis of the generated coding control information. 
[0191] The second recording medium also stores the 
above-mentioned coding control information giving in- 
struction not to change the display format of a video 
stream and a multiplexed stream containing the video 
stream on which a predetermined conversion process 
has been performed on the basis of the coding control 
information. 

[0192] Consequently, in any case, the reproduction 
side can prevent a mismatch from occurring between 
the video stream and the multimedia coding data. 
[0193] While the preferred embodiments of the 
present invention have been described using specific 
terms, such description is for illustrative purposes only, 
and it is to be understood that changes and variations 
may be made without departing from the scope of the 
appended claims. 

[0194] In so far as the embodiments of the invention 
described above are implemented, at least in part, using 
software-controlled data processing apparatus, it will be 
appreciated that a computer program providing such 
software control and a storage medium by which such 
a computer program is stored are envisaged as aspects 
of the present invention. 

[0195] Various different aspects and features of the 
present invention are defined in the appended claims. 
Combinations of features from the dependent claims 
may be combined with features of the independent 
claims as appropriate and not merely as explicitly set 
out in the claims. 



Claims 

1 . An image coding apparatus comprising: 

inputting means for inputting a multiplexed 
stream containing multimedia coding data; 
separating means for separating a video 
stream from said multiplexed stream inputted 
by said inputting means; 

converting means for performing a predeter- 
mined converting process on said video stream 
separated by said separating means; 
generating means for generating additional in- 
formation indicating that a mismatch will occur 
when the converted video stream is displayed 
on the basis of said multimedia coding data; 
and 

outputting means for outputting said converted 
video stream, said multimedia coding data, and 
said additional information. 



13 



BNSDOCID: <EP 1182880A2_L> 



25 



EP 1 182 880 A2 



2. The image coding apparatus acSP!Ting to claim 1 , 
further comprising: 

coding means for coding said additional infor- 
mation generated by said generating means as da- 
ta separate from said multiplexed stream containing 
said converted video stream. 

3. The image coding apparatus according to claim 1 , 
further comprising: 

coding means for multiplexing said additional 
information generated by said generating means 
with said multiplexed stream containing said con- 
verted video stream and then coding a multiplexed 
result. 

4. The image coding apparatus according to claim 1 , 
wherein said converting means converts a video 
stream picture frame parameter. 

5. The image coding apparatus according to claim 1 , 
wherein the conversion by said converting means 
includes at least a process of decoding the video 
stream separated by said separating means and a 
process of encoding the decoded video stream. 

6. The image coding apparatus according to claim 1 , 
wherein said additional information generated by 
said generating means contains at least one of orig- 
inal picture frame information and an original screen 
aspect ratio. 

7. The image coding apparatus according to claim 1, 
wherein said additional information generated by 
said generating means contains an original video 
format and a video format after the conversion. 

8. The image coding apparatus according to claim 1, 
wherein said additional information generated by 
said generating means contains an original screen 
aspect ratio and a screen aspect ratio after the con- 
version. 

9. The image coding apparatus according to claim 1, 
wherein said additional information generated by 
said generating means contains at least one of in- 
formation indicating whether or not a picture frame 
of said video stream has been converted by said 
converting means, information about an original 
picture frame of said video stream separated by 
said separating means, and an original screen as- 
pect ratio. 

10. An image coding method comprising the steps of: 

inputting a multiplexed stream containing mul- 
timedia coding data; 

separating a video stream from said multi- 
plexed stream inputted in said inputting step; 
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performing a predeTSmnined converting proc- 
ess on said video stream separated in said sep- 
arating step; 

generating additional information indicating 
that a mismatch will occur when the converted 
video stream is displayed on the basis of said 
multimedia coding data; 
and 

outputting said converted video stream, said 
multimedia coding data, and said additional in- 
formation. 

11. The image coding method according to claim 10, 
further comprising the step of: 

coding said additional information generated 
in said generating step as data separate from said 
multiplexed stream containing said converted video 
stream. 

12. The image coding method according to claim 10, 
further comprising: 

a coding step of multiplexing said additional 
information generated in said generating step with 
said multiplexed stream containing said converted 
video stream and then coding a multiplexed result. 

13. The image coding method according to claim 10, 
wherein a video stream picture frame parameter is 
converted in said converting step. 

14. The image coding method according to claim 10, 
wherein the conversion in said converting step in- 
cludes at least a process of decoding the video 
stream separated in said separating step and a 
process of encoding the decoded video stream. 

15. The image coding method according to claim 10, 
wherein said additional information generated in 
said generating step contains at least one of original 
picture frame information and an original screen as- 
pect ratio. 

16. The image coding method according to claim 10, 
wherein said additional information generated in 
said generating step contains an original video for- 
mat and a video format after the conversion. 

17. The image coding method according to claim 10, 
wherein said additional information generated in 
said generating step contains an original screen as- 
pect ratio and a screen aspect ratio after the con- 
version. 

18. The image coding method according to claim 10, 
wherein said additional information generated in 
said generating step contains at least one of infor- 
mation indicating whether or not a picture frame of 
said video stream has been converted in said con- 
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verting step, information abouf^Rriginal picture 
frame of said video stream separated in said sepa- 
rating step, and an original screen aspect ratio. 

19. A recording medium recorded a computer-readable 5 
program, said program comprising the steps of: 

inputting a multiplexed stream containing mul- 
timedia coding data; 

separating a video stream from said multi- 10 
plexed stream Inputted in said inputting step; 
performing a predetermined converting proc- 
ess on said video stream separated in said sep- 
arating step; 

generating additional information indicating 15 

that a mismatch will occur when the converted 

video stream is displayed on the basis of said 

multimedia coding data; and 

outputting said converted video stream, said 

multimedia coding data, and said additional in- 20 

formation. 

20. A recording medium recorded a video stream con- 
verted by a predetermined conversion process, 
multimedia coding data, and additional information 25 
indicating occurrence of a mismatch at displaying 
said converted video stream on the basis of said 
multimedia coding data. 

21. The recording medium according to claim 20, 30 
wherein said additional information is coded and re- 
corded as data different from a multiplexed stream 
containing said converted video stream. 

22. The recording medium according to claim 20, 35 
wherein said additional information is coded and re- 
corded as multiplexed with a multiplexed stream 
containing said converted video stream. 

23. The recording medium according to claim 20, 40 
wherein said video stream is converted in its picture 
frame parameter. 



24. The recording medium according to claim 20, 
wherein said video stream is decoded and then en- 
coded. 
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wherein said additional iWErmation contains infor- 
mation about an original screen aspect ratio and in- 
formation about a screen aspect ratio after the con- 
version. 

The recording medium according to claim 20, 
wherein said additional information contains at least 
one of information indicating whether or not a pic- 
ture frame of said video stream has been converted, 
information about an original picture frame of said 
video stream, and information about an original 
screen aspect ratio. 



29. An image decoding apparatus comprising: 

inputting means for inputting a multiplexed 
stream containing multimedia coding data; 
separating means for separating a video 
stream from said multiplexed stream inputted 
by said inputting means; 

decoding means for decoding said video 
stream separated by said separating means; 
and 

processing means for performing a predeter- 
mined conversion process on said decoded 
video stream in accordance with additional in- 
formation indicating occurrence of a mismatch 
at displaying said decoded video stream on the 
basis of said multimedia decoding data. 

30. The image decoding apparatus according to claim 
29, further comprising: 

acquiring means for acquiring said additional 
information from data different from said multi- 
plexed stream. 



31. 



The image decoding apparatus according to claim 
29, further comprising: 

acquiring means for acquiring said additional 
information from a multiplexed stream with which 
said additional information is multiplexed. 



45 



25. The recording medium according to claim 20, 
wherein said additional information contains at least 
one of original picture frame information and an so 
original screen aspect ratio. 

26. The recording medium according to claim 20, 
wherein said additional information contains infor- 
mation about an original video format and informa- 55 
tion about a video format after the conversion. 

27. The recording medium according to claim 20, 



32. The image decoding apparatus according to claim 
29, wherein said processing means converts a pic- 
ture frame parameter of said video stream. 

33. The image decoding apparatus according to claim 
29, wherein the conversion by said processing 
means includes at least a process of decoding said 
video stream separated by said separating means 
and a process of encoding the decoded video 
stream. 

34. The image decoding apparatus according to claim 
29, wherein said additional information contains at 
least one of information about an original picture 
frame and information about an original screen as- 
pect ratio. 
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35. The image decoding apparatus lUfSrding to claim 
29, wherein said additional information contains an 
original video format and information about a video 
format after the conversion. 

36. The image decoding apparatus according to claim 
29, wherein said additional information contains in- 
formation about an original screen aspect ratio and 
information about a screen aspect ratio after the 
conversion. 

37. The image decoding apparatus according to claim 
29, wherein said additional information contains at 
least one of information indicating whether or not a 
picture frame of said video stream has been con- 
verted by said converting means, information about 
an original picture frame of said video stream sep- 
arated by said separating means, and information 
about an original screen aspect ratio. 

38. An image decoding method comprising the steps of: 

inputting a multiplexed stream containing mul- 
timedia coding data; 

separating a video stream from said multi- 
plexed stream inputted in said inputting step; 
decoding said video stream separated in said 
separating step; and 

performing a predetermined conversion proc- 
ess on said decoded video stream in accord- 
ance with additional information indicating oc- 
currence of a mismatch at displaying said de- 
coded video stream on the basis of said multi- 
media decoding data. 

39. The image decoding method according to claim 38, 
further comprising the step of: 

acquiring said additional information from da- 
ta different from said multiplexed stream. 

40. The image decoding method according to claim 38, 
further comprising the step of: 

acquiring said additional information from a 
multiplexed stream with which said additional infor- 
mation is multiplexed. 

41. The image decoding method according to claim 38, 
wherein a picture frame parameter of said video 
stream is converted in said processing step. 

42. The image decoding method according to claim 38, 
wherein the conversion in said processing step in- 
cludes at least a process of decoding said video 
stream separated in said separating step and a 
process of encoding the decoded video stream. 

43. The image decoding method according to claim 38, 
wherein said additional information contains at least 
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one of information abourefi original picture frame 
and information about an original screen aspect ra- 
tio. 

44. The image decoding method according to claim 38, 
wherein said additional information contains an 
original video format and information about a video 
format after the conversion. 

45. The image decoding method according to claim 38, 
wherein said additional information contains infor- 
mation about an original screen aspect ratio and in- 
formation about a screen aspect ratio after the con- 
version. 

46. The image decoding method according to claim 38, 
wherein said additional information contains at least 
one of information indicating whether or not a pic- 
ture frame of said video stream has been converted 
in said converting step, information about an origi- 
nal picture frame of said video stream separated in 
said separating step, and information about an orig- 
inal screen aspect ratio. 

47. A recording medium recorded a computer-readable 
program, said program comprising the steps of: 

inputting a multiplexed stream containing mul- 
timedia coding data; 

separating a video stream from said multi- 
plexed stream inputted in said inputting step; 
decoding said video stream separated in said 
separating step; and 

performing a predetermined conversion proc- 
ess on said decoded video stream in accord- 
ance with additional information indicating oc- 
currence of a mismatch at displaying said de- 
coded video stream on the basis of said multi- 
media decoding data. 

48. An image coding apparatus comprising: 
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inputting means for inputting a multiplexed 
stream; 

separating means for separating a video 
stream from said multiplexed stream inputted 
by said inputting means; 

determining means for determining whether or 
not multimedia coding data is contained in said 
multiplexed stream inputted by said inputting 
means; 

generating means, if it is determined by said de- 
termining means that said multimedia coding 
data is contained in said multiplexed stream , for 
55 generating coding control information for in- 

structing not to change a display format of said 
video stream separated by said separating 
means; 
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converting means for perfo^pPTg a predeter- 
mined conversion process on said video 
stream separated by said separating means on 
the basis of said coding control information 
generated by said generating means; and 5 
multiplexing means for generating a multi- 
plexed stream which contains said video 
stream converted by said converting means. 

49. The image coding apparatus according to claim 48, 10 
wherein said generating means instructs not to 
change any of a picture frame, a video format, and 

an aspect ratio. 

50. An image coding method comprising the steps of: 15 

inputting a multiplexed stream; 
separating a video stream from said multi- 
plexed stream inputted in said inputting step; 
determining whether or not multimedia coding 20 
data is contained in said multiplexed stream in- 
putted in said inputting step; 
generating coding control information for in- 
structing not to change a display format of said 
video stream separated in said separating step, 25 
if it is determined in said determining step that 
said multimedia coding data is contained in said 
multiplexed stream; 

performing a predetermined conversion proc- 
ess on said video stream separated in said sep- 30 
arating step on the basis of said coding control 
information generated in said generating step; 
and 

generating a multiplexed stream which con- 
tains said video stream converted in said con- 35 
verting step. 



performing a prederWffmed conversion proc- 
ess on said video stream separated in said sep- 
arating step on the basis of said coding control 
information generated in said generating step; 
and 

generating a multiplexed stream which con- 
tains said video stream converted in said con- 
verting step. 

53. A recording medium recorded coding control infor- 
mation instructing not to change a display format of 
a video stream and a multiplexed stream containing 
a video stream on which a predetermined conver- 
sion process has been performed on the basis of 
said coding control information. 



51. The image coding method according to claim 50, 
wherein coding control information for instructing 
not to change any of a picture frame, a video format, 40 
and an aspect ratio is generated in the generating 
step. 



52. A recording medium recorded a computer-readable 

program, said program comprising the steps of: 45 

inputting a multiplexed stream; 
separating a video stream from said multi- 
plexed stream inputted in said inputting step; 
determining whether or not multimedia coding so 
data is contained in said multiplexed stream in- 
putted in said inputting step; 
generating coding control information for in- 
structing not to change a display format of said 
video stream separated in said separating step, 55 
if it is determined in said determining step that 
said multimedia coding data is contained in said 
multiplexed stream; 
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F IG.6 



MISMATCH FLAG 
mi smatch_MMi nf o_f I ag 



RE-ENCODED FLAG 
Re_encoded_f I ag 



FRAME SIZE CHANGE FLAG 
changed_f rame_s i ze_f I ag 



ORIGINAL HORIZONTAL SIZE 
or i g i na l_hor i zonta l_s i ze 



ORIGINAL VERTICAL SIZE 
o r i g i na I _ve r t i ca I _s i ze 



ORIGINAL DISPLAY ASPECT RATIO 
original _d i sp I ay_aspect_rat i o 



MULTIMEDIA 

DISPLAY 

SUB 

INFORMATION 
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F IG.7 



Syntax 


No. o : 
bits 


Mnemomic 


, „. 

rrogramlnfo U ( 






length 


32 


uimsbf 


reseved_for_word_a 1 ign 


8 


bslbf 


.in 

num_ot_program_sequences 


8 


uimsbf 


tor u=U, \ <num_of_program sequences", { 






SPN_program_sequences start 


32 


uimsbf 


progranrunap_PID 


16 


bslbf 


num_of_streams_in ps 


8 


uimsbf 


num_of_g roups 


8 


uimsbf 


f o r ( s tream_ /nc/ex=0 1 

s tream_ index<num_ of_s treams_ in_ps I 
stream_index±+) { 






stream.PID 


16 


u imsbf 


StreamCodinglnf o () 






} 






i f \num_of_groups>\) { 






i or u — Uf wnu/n^or groups 9 ( 






num_of_str earns in group 


8 


uimsbf I 


for (k=0;k<nu/ji_of streams in grouplk++) { 






stream index 


8 


uimsbf 


} 






if(num of streams in group%2==0) { 






reserved for word align 


8 


bslbf 


} 






[ ) 






} 






} 













EXAMPLE OF Programlnf o 0 SYNTAX 
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F IG.8 



Syntax 


No. of 
bits 


Mnemomic 


StreamCod i ng Info 0 { 






length 


8 


bslbf 


streajn_cod i ng_type 


8 


uimsbf 


i f (stream^coding^type^^Qy&Z) { 






video_f ormat 


4 


uimsbf 


frame_rate 


4 


uimsbf 


d i sp I ay_aspect_rat i o 


4 


uimsbf 


| reserved_for_word_al ign 


2 


bslbf 


cc_f lag 


1 


uimsbf 


or lginal_video_format flag 


1 




i f ( orig ina /_ video_ forma t flag*=\ ) { 






original_video format 


4 


uimsbf 


onginal_display aspect ratio 


4 


uimsbf 


reserved for word al ign 


8 


bslbf 


} 






} e 1 se if {stream__coding_ type==0x03// 
stream__coding_ type==0x04/ / 
s tream_ coding^ £ype== OxQFI I 
stream^coding^ type=-0x80l 7 
stream_coding type==0x8J) { 






audio_presentation type 


4 


uimsbf 


sampling frequency 


4 


uimsbf 


reserved for word align 


8 


bslbf 


} " " 






} 







StreamCod inglnfoO SYNTAX (ANOTHER EXAMPLE 
OF MULTIMEDIA DISPLAY SUB INFORMATION) 



.1182860A2_L> 
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FIG. 9 



st ream_cod i ng_type 



stream_cod i ng_type 


Mean i na 

if i w 111 1 ?4 


0x00-0x01 


reserved for future use 


0x02 


MPEG-1 or MPEG-2 video stream 


0x03 


MPEG-1 audio 


0x04 


MPEG-2 multi -channel audio, backward compatible to MPEG-1 


0x05 


reserved for future use 


0x06 


Teletext defined in SESF or DVB or 




Subtitle defined in ISDB 


0x07-0x09 


reserved for future use 


OxOA I 


IS0/IEC 13818-6 type A 


OxOB 


IS0/IEC 13818-6 type B 


OxOC 


IS0/IEC 13818-6 type C I 


OxOD 


I SO/ I EC 13818-6 type D 


OxOE 


reserved for future use 


OxOF 


MPEG-2 AAC audio with ADTS transport syntax j 


0x10-0x7F 


reserved for future use 


0x80 


SESF LPCM audio 


0x81 


Dolby AC-3 audio 


0x82-0xFF 


reserved for future use 
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F IG. 10 



video format 



video_format 


Mean i ng 


Video standard 


0 


480 i 


ITU-R BT. 601-4 


1 


576 i 


ITU-R BT. 601-4 


2 


480p 


SMPTE 293M 


3 


1080i 


SMPTE 274M 


4 


720p 


SMPTE 296M 


5-14 


reserved for future use 




15 


No information 





F IG. 1 1 



frame rate 



f rame_rate 


Meani ng 


0 


reserved for future use 


1 


24 000/1001 (23.976...) 


2 


24 


3 


25 


4 


30 000/1001 (29.97...) 


5 


30 


6 


50 


7 


60 000/1001 (59.94...) 


8 


60 


9-14 


reserved for future use 


15 


No information 



BNSDOCID: <EP 1 182880A2J_> 



27 



EP 1 182 880 A2 



F I G. 1 2 



d i sp I ay_aspect_rat i o 



d i sp I ay_aspect_rat i o 


Meaning 


0 


reserved for future use 


1 


reserved for future use 


2 


4*3 display aspect ratio 


3 


16:9 display aspect ratio 


4 


2.21M display aspect ratio 


5-14 


reserved for future use 


15 


No information 
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FIG. 13 



START CODING PROCESSING OF 
AV STREAM AND MULTIMEDIA 
DISPLAY SUB INFORMATION 



INPUT MULTIPLEXED 
STREAM INCLUDING 
MULTIMEDIA CODING DATA 



50 



SEPARATE VIDEO STREAM 
FROM MULTIPLEXED STREAM 



RE-ENCODE VIDEO STREAM 



MULTIPLEX ABOVE- 
MENTIONED VIDEO STREAM 
WITH MULTIMEDIA CODING 
DATA TO GENERATE 
MULTIPLEXED STREAM 



51 



52 



53 



GENERATE MULTIMEDIA 
DISPLAY SUB INFORMATION 



54 



(END) 
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F IG. 14 



START CODING PROCESSING FOR 
RESTRICTING RE -ENCODING OF VIDEO 
OF MULTIPLEXED STREAM INCLUDING 
MULTIMEDIA CODING DATA 




INPUT MULTIPLEXED STREAM 



SEPARATE VIDEO STREAM 
FROM MULTIPLEXED STREAM 



70 
71 



IF MULTIMEDIA CODING DATA 
IS INCLUDED. RE-ENCODE 
VIDEO STREAM WITHOUT 
CHANGING VIDEO DISPLAY 
FORMAT 



72 



GENERATE MULTIPLEXED 
STREAM INCLUDING ABOVE- 
MENTIONED VIDEO STREAM 



73 



' r_ 

(end) 



30 




ccz: 



tu_oo 

CQOI- 
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F I G. 1 7 



START CONTROL PROCESSING 
WITH VIDEO RE-ENCODING 
ALLOCATED BIT RATE FIXED 



3 



SET RECORDING RATE S 



S1 



DETERMINE NON-VIDEO 
STREAMS TO BE RECORDED 
AND COMPUTE MAXIMUM 
TOTAL VALUE D OF THEIR 
BIT RATES 



S2 



USE C = S-D AS VIDEO 
RE-ENCODING ALLOCATED 
BIT RATE 



S3 



ANALYZE CODING INFORMATION 
OF INPUT TS VIDEO STREAM 



I 



S4 



ON THE BASIS OF VALUE C 
AND RESULT OF ANALYSIS 
OF CODING INFORMATION, 
DETERMINE VIDEO CODING 
PARAMETERS SUCH THAT 
OPTIMUM PICTURE QUALITY 
IS OBTAINED 



S5 





i 


RE 


-ENCODE 


VIDEO 


DATA 


OF 


CURRENT UNIT 


TIME 



S6 




(end) 
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F I G. 1 8 



START CONTROL PROCESSING 
WITH VIDEO RE-ENCODING 
ALLOCATED BIT RATE VARIABLE 



SET RECORDING RATE S 



S21 



ANALYZE CODING INFORMATION 
OF INPUT TS VIDEO STREAM 



S22 



COMPUTE TOTAL BIT RATE 
B OF PREDETERMINED flNIT 
TIME OF STREAMS OTHER 
THAN INPUT TS VIDEO 



S23 



USE C = S-B AS VIDEO 
RE-ENCODING ALLOCATED 
BIT RATE 



S24 



ON THE BASIS OF VALUE C 
AND RESULT OF ANALYSIS 
OF CODING INFORMATION, 
DETERMINE VIDEO CODING 
PARAMETERS SUCH THAT 
OPTIMUM PICTURE QUALITY 
IS OBTAINED 



S25 



V 


RE 


-ENCODE VIDEO 


DATA 


OF 


CURRENT UNIT 


TIME 



S26 
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FIG. 24 



START AV STREAM REPRODUCTION 
PROCESSING USING MULTIMEDIA 
DISPLAY SUB INFORMATION 



INPUT MULTIPLEXED 
STREAM CONTAINING 
MULTIMEDIA CODING DATA 



INPUT MULTIMEDIA DISPLAY 
SUB INFORMATION 



SEPARATE VIDEO STREAM 
FROM MULTIPLEXED STREAM 



DECODE VIDEO STREAM 



IF THERE EXISTS MISMATCH 
BETWEEN VIDEO AND 
MULTIMEDIA CODING DATA, 
SCALE-CONVERT VIDEO ON 
THE BASIS OF MULTIMEDIA 
DISPLAY SUB INFORMATION 



60 



61 



62 



63 



64 





i 


SYNTHESIZE P 
IMAGE AND MUI 
TO CREATE DIJ 


ROCESSED 
-TIMEDIA DATA 
SPLAY IMAGE 



65 



(end) 
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